Skip to content

Add finance grounding verification - #9

Merged
psgundecha-nv merged 4 commits into
NVIDIA:mainfrom
aalvsz:provenanceguard-finance-v2
Aug 24, 2026
Merged

Add finance grounding verification#9
psgundecha-nv merged 4 commits into
NVIDIA:mainfrom
aalvsz:provenanceguard-finance-v2

Conversation

@aalvsz

@aalvsz aalvsz commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a post-rollout GroundingVerifier stage that checks finance-agent answers against SEC retrieval excerpts and writes separate allow/block/unavailable decisions. It does not modify rewards, training data, or original rollouts.

It verifies source-bound calculations, comparisons, and evidence-scoped refusals, including entity, metric, date, value, and filing attribution.

Held-out evaluation

  • 180 native finance-agent rollouts across 60 unseen issuers
  • 120 frozen eligible traces: 40 calculations, 40 comparisons, 40 refusals
  • grounded acceptance: 119/120 (99.17%)
  • adversarial rejection: 120/120 (100%)
  • zero false allows

Attacks cover numeric fabrication; entity, metric, temporal, and source conflation; comparison errors; and unsupported refusal claims.

See the evaluation protocol and results.

Validation

  • 385 tests passed; 1 model-gated test skipped in the default run
  • pinned-public-model integration gate passed
  • Ruff, lock, license-header, and diff checks passed

Scope

The verifier checks retrieval excerpts retained in native NVFlow traces. It does not independently validate complete SEC filings or establish unrestricted production hallucination detection.

@aalvsz
aalvsz requested a review from a team August 11, 2026 21:58
@aalvsz
aalvsz force-pushed the provenanceguard-finance-v2 branch from cb63d9f to 8211530 Compare August 13, 2026 07:44
Signed-off-by: Ander Alvarez Sanz <104446704+aalvsz@users.noreply.github.com>
@aalvsz
aalvsz force-pushed the provenanceguard-finance-v2 branch from ed6fe5c to 055905a Compare August 13, 2026 08:33
@aalvsz aalvsz changed the title Add finance provenance evaluation Add finance grounding verification Aug 13, 2026
This was referenced Aug 13, 2026
Comment thread docs/recipes/finance/grounding-verifier/README.md
@sgharghabi

Copy link
Copy Markdown

Thanks @aalvsz for your contribution.
if I understand correctly your pipeline:

  1. Splits a finance agent's rollout answer into individual claims
  2. Matches each claim to its specific evidence that was retrieved during the rollout
  3. Runs an NLI entailment check, classifies each claim as entailment (supported), contradiction, or neutral relative to its matched evidence, using the DeBERTa NLI model
  4. Runs strict deterministic checks (exact numbers/dates, entity/source conflation, metric mismatches) as a backstop, since the NLI model alone can be fooled by fluent-but-wrong numbers

That is interesting and can be used as an independent feature that validate grpo rollouts.
I am about to run your pipeline and if everything works properly. The rest looks good to me.

Comment thread docs/recipes/finance/grounding-verifier/evaluation-protocol.md
aalvsz added 2 commits August 15, 2026 08:01
Signed-off-by: Ander Alvarez Sanz <104446704+aalvsz@users.noreply.github.com>
Signed-off-by: Ander Alvarez Sanz <104446704+aalvsz@users.noreply.github.com>
@aalvsz

aalvsz commented Aug 15, 2026

Copy link
Copy Markdown
Contributor Author

@sgharghabi

That is interesting and can be used as an independent feature that validate grpo rollouts. I am about to run your pipeline and if everything works properly. The rest looks good to me.

You understood the pipeline perfectly. For now deterministic claim decomposition works fine, but this step could be made more accurate without introducing excessive latency (looking into it) . Then MiniLM is used only to route each claim to evidence, and DeBERTa performs the NLI check. The final decision is made after running some deterministic checks like you mentioned.

@sgharghabi

Copy link
Copy Markdown

@aalvsz it seems two Code linting and Unit tests are failing, Would you please fix it?

Signed-off-by: Ander Alvarez Sanz <104446704+aalvsz@users.noreply.github.com>
@sgharghabi
sgharghabi self-requested a review August 22, 2026 18:34

@psgundecha-nv psgundecha-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sgharghabi has successfully tested end-2-end.

@psgundecha-nv
psgundecha-nv merged commit 382a128 into NVIDIA:main Aug 24, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants